1. Field of the Invention
The present invention relates to an image processing technique for restoring an image deteriorated due to an influence of an optical system to produce a high-resolution image.
2. Description of the Related Art
In image capturing of an object through an optical system, light emitted from one point of the object may spatially spread without converging to one point on an image plane, due to influences of diffraction caused by the optical system and aberration of the optical system. Such spread is expressed by a PSF (Point Spread Function). And, in the image acquired by the image capturing (by using an image sensor), an image blur component is generated by convolving a spread component expressed by the PSF to an object area, which causes deterioration of image resolution.
As a method of correcting such image deterioration due to the optical system by using an image processing technique (image restoration process), the following method is proposed.
When, in a real space (x,y), f(x,y) represents a non-deteriorated image, the PSF represents h(x,y), g(x,y) represents a deteriorated image and “*” represents convolution, the following expression is established:g(x,y)=∫∫f(X,Y)*h(x−X,y−Y)dXdY  (1).
When performing Fourier transform on the expression (1) to transform it from the real space (x,y) to a frequency space (u,v), the following expression is obtained:G(u,v)=F(u,v)×H(u,v)  (2)where F(u,v) represents a result of the Fourier transform of f(x,y), G(u,v) represents a result of the Fourier transform of g(x,y), and H(u,v) represents a result of the Fourier transform of h(x,y).
Therefore, the following expression is obtained:F(u,v)=G(u,v)/H(u,v)  (3).
As understood from the expression (3), dividing G(u,v) that is the result of the Fourier transform of the deteriorated image g(x,y) by H(u,v) that is the result of the Fourier transform of the PSF h(x,y) in the frequency space can provide F(u,v) that is the result of the Fourier transform of the non-deteriorated image f(x,y). Therefore, performing inverse Fourier transform on the F(u,v) enables acquisition of the non-deteriorated image (restored image) f(x,y).
However, acquiring the restored image by such an image restoration process greatly amplifies a noise component generated by the image sensor, which makes it difficult to acquire a good image. In this regard, there is known an image restoration process using a Wiener filter, which is expressed by the following expression (4), for suppressing such noise amplification:1/H(u,v)×|H(u,v)|2/(|H(u,v)|2+┌)  (4),
where H(u,v) represents an OTF (Optical Transfer Function), and ┌ represents a constant to reduce an amplification amount of the noise component.
Multiplying an OTF having frequency information and phase information of the optical system by the expression (4) makes a phase of the PSF generated due to diffraction or aberration of the optical system zero and amplifies a frequency characteristic, which makes it possible to obtain a good image with a high resolution.
Moreover, Japanese Patent Laid-Open No. 2006-238032 discloses an image processing method that performs an image restoration process with setting a slight spread of a PSF after the image restoration process. Furthermore, Japanese Patent Laid-Open No. 2009-124568 discloses an image restoration process that restores a deteriorated image by using an image restoration filter produced from a PSF.
In the image pickup apparatus, the object image (optical image) formed by the optical system is electrically sampled by the image sensor (image pickup element) such as a CCD (Charge Coupled Device) and a CMOS (Complementary Metal Oxide Semiconductor Image Sensor). When being sampled by the image sensor, the optical image that is originally a continuous quantity is converted into discrete values by plural pixels of the image sensor, and thereby a frequency signal whose cycle corresponding to a sampling frequency is produced in the frequency space. Due to the cycle, when the frequency signal distributes over ½ of the sampling frequency, a frequency component of the frequency signal over the ½ of the sampling frequency is folded to generate an alias signal (or aliasing or folded signal), which makes it impossible to produce an accurate image signal. The frequency that is ½ of the sampling frequency is called a Nyquist frequency.
In general, significant amplification of a high frequency component in the image restoration process for restoring the deteriorated image amplifies not only the noise component but also the alias signal having frequencies over the Nyquist frequency, which generates moire, coloring, artifact or the like in the restored image.